Predictability effects in auditory scene analysis: a review

Many sound sources emit signals in a predictable manner. The idea that predictability can be exploited to support the segregation of one source's signal emissions from the overlapping signals of other sources has been expressed for a long time. Yet experimental evidence for a strong role of predictability within auditory scene analysis (ASA) has been scarce. Recently, there has been an upsurge in experimental and theoretical work on this topic resulting from fundamental changes in our perspective on how the brain extracts predictability from series of sensory events. Based on effortless predictive processing in the auditory system, it becomes more plausible that predictability would be available as a cue for sound source decomposition. In the present contribution, empirical evidence for such a role of predictability in ASA will be reviewed. It will be shown that predictability affects ASA both when it is present in the sound source of interest (perceptual foreground) and when it is present in other sound sources that the listener wishes to ignore (perceptual background). First evidence pointing toward age-related impairments in the latter capacity will be addressed. Moreover, it will be illustrated how effects of predictability can be shown by means of objective listening tests as well as by subjective report procedures, with the latter approach typically exploiting the multi-stable nature of auditory perception. Critical aspects of study design will be delineated to ensure that predictability effects can be unambiguously interpreted. Possible mechanisms for a functional role of predictability within ASA will be discussed, and an analogy with the old-plus-new heuristic for grouping simultaneous acoustic signals will be suggested

Sound predictability as a higher-order cue in auditory scene analysis

A major challenge for the auditory system is to disentangle signals emitted by two or more sound sources that are active in a temporally interleaved manner (sequential stream segregation). Besides distinct characteristics of the individual signals (e.g., their timbre, location, and pitch), one important cue for distinguishing the sound sources is how their emitted signals unfold over time. It seems intuitively plausible that signals that unfold predictably with respect to their acoustic features and time-points of occurrence, such as the repetitive signature of a train moving on the rails, can be more readily identified as originating from one sound source. Based on this rationale, predictive elements have successfully been incorporated into computational models of auditory scene analysis for many years

Modulation-frequency acts as a primary cue for auditory stream segregation

In our surrounding acoustic world sounds are produced by different sources and interfere with each other before arriving to the ears. A key function of the auditory system is to provide consistent and robust descriptions of the coherent sound groupings and sequences (auditory objects), which likely correspond to the various sound sources in the environment. This function has been termed auditory stream segregation. In the current study we tested the effects of separation in the frequency of amplitude modulation on the segregation of concurrent sound sequences in the auditory stream-segregation paradigm (van Noorden 1975). The aim of the study was to assess 1) whether differential amplitude modulation would help in separating concurrent sound sequences and 2) whether this cue would interact with previously studied static cues (carrier frequency and location difference) in segregating concurrent streams of sound. We found that amplitude modulation difference is utilized as a primary cue for the stream segregation and it interacts with other primary cues such as frequency and location difference

Using binocular rivalry to tag foreground sounds: Towards an objective visual measure for auditory multistability

In binocular rivalry, paradigms have been proposed for unobtrusive moment-by-moment readout of observers' perceptual experience (“no-report paradigms”). Here, we take a first step to extend this concept to auditory multistability. Observers continuously reported which of two concurrent tone sequences they perceived in the foreground: high-pitch (1008 Hz) or low-pitch (400 Hz) tones. Interstimulus intervals were either fixed per sequence (Experiments 1 and 2) or random with tones alternating (Experiment 3). A horizontally drifting grating was presented to each eye; to induce binocular rivalry, gratings had distinct colors and motion directions. To associate each grating with one tone sequence, a pattern on the grating jumped vertically whenever the respective tone occurred. We found that the direction of the optokinetic nystagmus (OKN)—induced by the visually dominant grating—could be used to decode the tone (high/low) that was perceived in the foreground well above chance. This OKN-based readout improved after observers had gained experience with the auditory task (Experiments 1 and 2) and for simpler auditory tasks (Experiment 3). We found no evidence that the visual stimulus affected auditory multistability. Although decoding performance is still far from perfect, our paradigm may eventually provide a continuous estimate of the currently dominant percept in auditory multistability

Feature predictability flexibly supports auditory stream segregation or integration

Many sound sources emit series of discrete sounds. Auditory perception must bind these sounds together (stream integration) while separating them from sounds emitted by other sources (stream segregation). One cue for identifying successive sounds that belong together is the predictability between their feature values. Previous studies have demonstrated that independent predictable patterns appearing separately in two interleaved sound sequences support perceptual segregation. The converse case, whether a joint predictable pattern in a mixture of interleaved sequences supports perceptual integration, has not yet been put to a rigorous empirical test. This was mainly due to difficulties in manipulating the predictability of the full sequence independently of the predictability of the interleaved subsequences. The present study implemented such an independent manipulation. Listeners continuously indicated whether they perceived a tone sequence as integrated or segregated, while predictable patterns set up to support one or the other percept were manipulated without the participants’ knowledge. Perceptual reports demonstrate that predictability supports stream segregation or integration depending on the type of predictable pattern that is present in the sequence. The effects of predictability were so pronounced as to qualitatively flip perception from predominantly (62%) integrated to predominantly (73%) segregated. These results suggest that auditory perception flexibly responds to encountered regular patterns, favoring predictable perceptual organizations over unpredictable ones. Besides underlining the role of predictability as a cue within auditory scene analysis, the present design also provides a general framework that accommodates previous investigations focusing on sub-comparisons within the present set of experimental manipulations. Results of intermediate conditions shed light on why some previous studies have obtained little to no effects of predictability on auditory scene analysis

Different roles of similarity and predictability in auditory stream segregation

Sound sources often emit trains of discrete sounds, such as a series of footsteps. Previously, two dif¬ferent principles have been suggested for how the human auditory system binds discrete sounds to¬gether into perceptual units. The feature similarity principle is based on linking sounds with similar characteristics over time. The predictability principle is based on linking sounds that follow each other in a predictable manner. The present study compared the effects of these two principles. Participants were presented with tone sequences and instructed to continuously indicate whether they perceived a single coherent sequence or two concurrent streams of sound. We investigated the influence of separate manipulations of similarity and predictability on these perceptual reports. Both grouping principles affected perception of the tone sequences, albeit with different characteristics. In particular, results suggest that whereas predictability is only analyzed for the currently perceived sound organization, feature similarity is also analyzed for alternative groupings of sound. Moreover, changing similarity or predictability within an ongoing sound sequence led to markedly different dynamic effects. Taken together, these results provide evidence for different roles of similarity and predictability in auditory scene analysis, suggesting that forming auditory stream representations and competition between alter¬natives rely on partly different processes

Effects of multiple congruent cues on concurrent sound segregation during passive and active listening: An event-related potential (ERP) study

In two experiments, we assessed the effects of combining different cues of concurrent sound segregation on the object-related negativity (ORN) and the P400 event-related potential components. Participants were presented with sequences of complex tones, half of which contained some manipulation: One or two harmonic partials were mistuned, delayed, or presented from a different location than the rest. In separate conditions, one, two, or three of these manipulations were combined. Participants watched a silent movie (passive listening) or reported after each tone whether they perceived one or two concurrent sounds (active listening). ORN was found in almost all conditions except for location difference alone during passive listening. Combining several cues or manipulating more than one partial consistently led to sub-additive effects on the ORN amplitude. These results support the view that ORN reflects an integrated, feature-unspecific assessment of the auditory system regarding the contribution of two sources to the incoming sound

Regularity extraction from non-adjacent sounds

The regular behavior of sound sources helps us to make sense of the auditory environment. Regular patterns may, for instance, convey information on the identity of a sound source (such as the acoustic signature of a train moving on the rails). Yet typically, this signature overlaps in time with signals emitted from other sound sources. It is generally assumed that auditory regularity extraction cannot operate upon this mixture of signals because it only finds regularities between adjacent sounds. In this view, the auditory environment would be grouped into separate entities by means of readily available acoustic cues such as separation in frequency and location. Regularity extraction processes would then operate upon the resulting groups. Our new experimental evidence challenges this view. We presented two interleaved sound sequences which overlapped in frequency range and shared all acoustic parameters. The sequences only differed in their underlying regular patterns. We inserted deviants into one of the sequences to probe whether the regularity was extracted. In the first experiment, we found that these deviants elicited the mismatch negativity (MMN) component. Thus the auditory system was able to find the regularity between the non-adjacent sounds. Regularity extraction was not influenced by sequence cohesiveness as manipulated by the relative duration of tones and silent inter-tone-intervals. In the second experiment, we showed that a regularity connecting non-adjacent sounds was discovered only when the intervening sequence also contained a regular pattern, but not when the intervening sounds were randomly varying. This suggests that separate regular patterns are available to the auditory system as a cue for identifying signals coming from distinct sound sources. Thus auditory regularity extraction is not necessarily confined to a processing stage after initial sound grouping, but may precede grouping when other acoustic cues are unavailable

Regularity Extraction from Non-Adjacent Sounds

The regular behavior of sound sources helps us to make sense of the auditory environment. Regular patterns may, for instance, convey information on the identity of a sound source (such as the acoustic signature of a train moving on the rails). Yet typically, this signature overlaps in time with signals emitted from other sound sources. It is generally assumed that auditory regularity extraction cannot operate upon this mixture of signals because it only finds regularities between adjacent sounds. In this view, the auditory environment would be grouped into separate entities by means of readily available acoustic cues such as separation in frequency and location. Regularity extraction processes would then operate upon the resulting groups. Our new experimental evidence challenges this view. We presented two interleaved sound sequences which overlapped in frequency range and shared all acoustic parameters. The sequences only differed in their underlying regular patterns. We inserted deviants into one of the sequences to probe whether the regularity was extracted. In the first experiment, we found that these deviants elicited the mismatch negativity (MMN) component. Thus the auditory system was able to find the regularity between the non-adjacent sounds. Regularity extraction was not influenced by sequence cohesiveness as manipulated by the relative duration of tones and silent inter-tone-intervals. In the second experiment, we showed that a regularity connecting non-adjacent sounds was discovered only when the intervening sequence also contained a regular pattern, but not when the intervening sounds were randomly varying. This suggests that separate regular patterns are available to the auditory system as a cue for identifying signals coming from distinct sound sources. Thus auditory regularity extraction is not necessarily confined to a processing stage after initial sound grouping, but may precede grouping when other acoustic cues are unavailable

Neuronal adaptation, novelty detection and regularity encoding in audition

The ability to detect unexpected stimuli in the acoustic environment and determine their behavioral relevance to plan an appropriate reaction is critical for survival. This perspective article brings together several viewpoints and discusses current advances in understanding the mechanisms the auditory system implements to extract relevant information from incoming inputs and to identify unexpected events. This extraordinary sensitivity relies on the capacity to codify acoustic regularities, and is based on encoding properties that are present as early as the auditory midbrain. We review state-of-the-art studies on the processing of stimulus changes using non-invasive methods to record the summed electrical potentials in humans, and those that examine single-neuron responses in animal models. Human data will be based on mismatch negativity (MMN) and enhanced middle latency responses (MLR). Animal data will be based on the activity of single neurons at the cortical and subcortical levels, relating selective responses to novel stimuli to the MMN and to stimulus-specific neural adaptation (SSA). Theoretical models of the neural mechanisms that could create SSA and novelty responses will also be discussed